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Power — what  is  it? 
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Truth 


Decision 


H0:  No  Change 

HA^hange 

H0:  No  Change 

Correct  Negative 

False  Positive 

Ha:  Change 

False  Negative 

Correct  Positive 

•  Power  =  Pr(HA 

J  HA  is  true) 

-  Choose  sample  size,  n,  to  get  this 

-  Also  need  to  decide  what  you  want  to  see... stay 
tuned... 


•  a  =  Pr(HA  |  H0  is  true) 

-  Choose  this  number  directly 

-  Normally  0.05  or  0.1 
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Svf  Uncertainty  and  its  vicissitudes 
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o 


o 


o 


The  mean  is  never  equal  to  0. 
But  can  we  see  the  difference 
given  how  much 
spread/uncertainty  there  is  in 
the  sample? 
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Svf  Uncertainty  and  its  vicissitudes 
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The  confidence  intervals  put  a 
measure  on  uncertainty  to  help 
us  make  a  decision. 

Here  a  difference  is  not 
detected  because  the 
confidence  intervals  cross  zero. 
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Again... Power — what  is  it? 


AFFTC 


•  Power  is  the  proportion  of  times,  in  the  long 
run,  that  our  test  (t-test,  Cl)  identifies  a 
difference,  when  it  really  exists. 


•  WRT  the  mean,  we  need  to  decide  how  big  of  a 
difference  from  zero  do  we  care  about. 

-  This  is  the  effect  size,  called  5. 

•  If  we  choose  enough  samples  the  Cl  shrinks, 
and  it  is  easier  to  see  a  difference. 

•  But  how  large  of  a  sample  do  we  need  to  get  to 
see  the  5  we  want? 


Power — via  Monte  Carlo 
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•  For  many  applications,  such  as  the  one  given, 
power  calculations  are  closed  form. 

•  For  other  difficult  applications,  the  calculation 
doesn"texist. 

•  Generate  Alternate  Population 

-  With  the  desired  effect 

-  With  the  distribution  characteristics  needed 

•  Sample  from  it  repeatedly 

-  Each  time  analyzing  the  sample  and  record 
significance 

•  Compute  the  proportion  of  significant  outcomes 

-  This  is  power  via  Monte  Carlo 


Setting  Up  the  „Mhimal"  Alternative 

(1 -sample  t-test) 
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Effect  Size  5 


Couch  the  example  in 
a  one-sample  t-test 
for  simplicity’s  sake. 


We  want  to  see  at  least  a  5 
effect  size  80%  of  the 
time... 
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;  Setting  Up  the  „Mhimal"  Alternative 

(1 -sample  t-test) 
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Couch  the  example  in 
a  one-sample  t-test 
for  simplicity’s  sake. 


With  this  much 
uncertainty 
(characteristics  of 
population). 


-4 

We  want  to  see  at  least  a  5 
effect  size  80%  of  the 


-2  Q _ _  5  2  4 

Alternate  Hypothetical  Population 


time... 
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Using  the  Alternate  World 
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Couch  the  example  in 
a  one-sample  t-test 
for  simplicity’s  sake. 

Now  repeatedly 
sample  from  the 
population  and  run  t- 
tests  each  time. 

Each  sample  will 
have  a  different 
estimate  for  mean 
and  standard 
deviation. 


With  this  much 
uncertainty 
(characteristics  of 
population). 


Effect  Size  5  and 


n  =  10 


o 

o 

o 


Sample  1 
Sample  2 
Sample  3 
Sample  4 


95%  Cl 

Sample  1 

0.45 

1.72 

Sample  2 

-0.02 

1.81 

Sample  3 

0.25 

2.08 

Sample  4 

0.39 

1.85 

Sample  1000 

0.41 

1.59 

We  want  to  see  a  5  effect 
size  80%  of  the  time. . . 


al  Alternate  Hypothetical  Population 
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Power  Estimate 
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•  With  n=10,  st.dev=1,  5=1  for  Normal  distribution 
we  get — 

•  Using  the  Monte  Carlo  method  to  calculate  power 
for  the  one-sample  t-test: 

-  Power  =  80.2% 

-  This  method  has  a  little  variation  in  the  estimate 
because  it  is  a  simulation  approach. 

•  Using  conventional  methods: 

-  Power  =  80.31% 
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Click  through  PDF 
“SerialMeans.pdf”  File 


SERIAL  CORRELATION 


Means  from  Different  Samples 


2 -Sam pie  t-test  w /  Serial  Corr. 
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Incorrect  SE  of  mean  Actual  SE  of  mean. 


Mean(X)  Mean(Y)  Mean(X)  Mean(Y) 

AR(1)  AR(1)  Independent  Independent 
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W  Need  a  different  version  of  test. 
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•  The  difference  in  a  regular  2-sample  t-test,  and 
an  adjusted  test  is, 

-  Estimate  the  autocorrelation,  r 

-  Adjust  the  SE  of  the  test  statistic: 


SEQ x  -  yU,,,,*  =  SE(x  -  50  * 

Cl  =  x-y±  zx_ySE(x  -  y)adjusted 


What  is  the  conventional  method  of  computing 
power  for  this? 
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How  do  we  do  it? 

^ -  AFFTC 

1 .  Create  the  minimal  alternate  hypothetical 
population  MAHP 

2.  Take  sample  of  size  n  from  the  MAHP 

3.  Test  to  see  significance  with  chosen  test,  (here 
we"reusing  the  adjusted  Cl  previous  page). 

4.  Repeat  steps  2  and  3  1000, 10000,  or  more  times 
while  recording  how  many  are  significant. 

5.  Find  proportion  that  are  significant  out  of 
number  of  repeated  loops. 
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Method:  Plug  and  Play 
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Question:  What  n  do  we  use  to  detect  8  with  chosen  power? 


r 


~\ 


1.  Generate  MAHP 

Minimal  Alternate  Hypothetical  Population 
Size  100,000  (or  something  real  big) 


There  may  be  multiple 
ways  of  testing  effect. 
•Method  1 
•Method  2 
•Method  3 

Which  is  most  powerful 
for  given  sample  size? 


2.  Sample  n  values  from  MAHP 


3.  Test  effect  with  method 


Repeat  loop 
IK  or  10K 
times 


Power  = 


1  1000 

— y 

iooo  tr 


v, 


4.  Record  outcome  from 
test  (significant  or  not) 


Get  vector,  v,  of  0/1,  1 
for  significant  outcome 
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CE90  Power 
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•  We  want  to  know  how  many  runs  we  need  to 
prove  CE90  is  meeting  spec  for  a  targeting 
device. 

-  How  close  to  spec  do  you  want  to  be  before  you 
are  willing  to  concede  that  you  are  no  different 
from  spec? 

-  We  want  to  have  proof  of  meeting  spec  if  it  is  at 
least  2  feet  beyond  CE90. 
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CE90  MAHP 
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CE90  MAHP 


w 

♦A* 


CE90 


-20  -10  0  10  20 
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AFFK 
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CE90  MAHP 


Jt 


*#* 


AFFK 


> 


o 

C\J 


o 


o 

T 


n=20 

Sample  1 
o  Sample  2 
Sample  3 
o  Sample  4 


CE90 
CE90  +  5 
MAHP 


-20  -10  0  10  20 


X 
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CE90  Power 


v 

♦A* 


AFFTC 


•  With  5=2  and  sample  size  of  60, 

-  Power  =  70% 

*  This  power  calculation  is  only  for  the  specific 
situations  similar  to  the  MAHP.  Any  different 
pattern  in  CE  will  require  a  separate  power 
analysis. 
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Summary 
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•  Monte  Carlo  power  estimation  is  versatile  and 
can  handle  most  situations. 


*  It  is  difficult  sometimes  to  create  the  MAHP. 


*  A  statistician  is  likely  needed  to  aid  in  the 
process. 
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